Skip to content

Add object streams and cross-reference streams to the writer - #7

Merged
tannevaled merged 1 commit into
mainfrom
object-streams
Aug 25, 2026
Merged

Add object streams and cross-reference streams to the writer#7
tannevaled merged 1 commit into
mainfrom
object-streams

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

A PDF is mostly small dictionaries, and a dictionary written on its own cannot
be compressed at all — there is nowhere to say how. Object streams are the
format's answer: pack many objects into one stream and compress that. This
adds them, and the cross-reference stream that is the only form able to say
where a packed object lives.

NewPackedWriter is the same Writer with that turned on. Objects of generation
zero that are not themselves streams wait to be gathered up; streams are
written where they stand, and one arriving with no filter of its own is
deflated, since nothing this package generates comes compressed and a packed
file is being written to be small. Two hundred objects go in each stream — one
has to be inflated whole to read any object in it, so packing everything into
one would make opening a file read all of it. The header is raised to 1.5, the
version that introduced this, which is 2003.

Measured on the corpus: all 118 833 files still rewrite to an identical
fingerprint, and the output falls from 98.8% of the input to 94.6% — about 1.9
GB across the corpus. It also settles the 355 files that were coming out larger
than they went in.

The reader has been able to read both forms since its second wave, so this is
the writer catching up with it — and the two now check each other on every file
in the corpus.

Exact 100% statement coverage, go vet clean, nine cross-compilation targets.

A PDF is mostly small dictionaries, and a dictionary written on its own cannot
be compressed at all — there is nowhere to say how. Object streams are the
format's answer: pack many objects into one stream and compress that. This
adds them, and the cross-reference stream that is the only form able to say
where a packed object lives.

NewPackedWriter is the same Writer with that turned on. Objects of generation
zero that are not themselves streams wait to be gathered up; streams are
written where they stand, and one arriving with no filter of its own is
deflated, since nothing this package generates comes compressed and a packed
file is being written to be small. Two hundred objects go in each stream — one
has to be inflated whole to read any object in it, so packing everything into
one would make opening a file read all of it. The header is raised to 1.5, the
version that introduced this, which is 2003.

Measured on the corpus: all 118 833 files still rewrite to an identical
fingerprint, and the output falls from 98.8% of the input to 94.6% — about 1.9
GB across the corpus. It also settles the 355 files that were coming out larger
than they went in.

The reader has been able to read both forms since its second wave, so this is
the writer catching up with it — and the two now check each other on every file
in the corpus.

Exact 100% statement coverage, go vet clean, nine cross-compilation targets.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 1ad8035 into main Aug 25, 2026
1 check passed
@tannevaled
tannevaled deleted the object-streams branch August 25, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant